SQL: Auto union Iceberg and Redpanda topic#575
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
03d3421 to
3fd1c5a
Compare
44f89d0 to
72c11f6
Compare
| warehouse = 's3://lakehouse-data/', | ||
| auth_type = 'oauth2', | ||
| oauth2_client_id = '<client-id>', | ||
| oauth2_client_secret = '<client-secret>', |
There was a problem hiding this comment.
Would we tell users to create and reference secrets here in the same way we do for RP Cloud secrets such as for catalog credentials in the cluster config, like so? https://deploy-preview-575--rp-cloud.netlify.app/redpanda-cloud/manage/iceberg/use-iceberg-catalogs/#use-a-secret-in-cluster-configuration
|
|
||
| // TODO: SME — confirm when REFRESH must be run on the linked Iceberg table. Source shows that if the Iceberg table's schema isn't refreshed, the query fails at planning time with: `Schema not found for Iceberg table '<table>'. Run: REFRESH <catalog>=><table>`. Confirm: | ||
| // - Is REFRESH required only the first time, or every time the Iceberg schema changes? | ||
| // - Is REFRESH required when new records are added to the Iceberg table (no schema change), or only on schema change? |
There was a problem hiding this comment.
REFRESH pertains to the schema/shape of the table only. The whole point of the bridge queries is to fetch all data without refreshing anything.
| == Query live and historical records together | ||
|
|
||
| // TODO: SME — confirm when REFRESH must be run on the linked Iceberg table. Source shows that if the Iceberg table's schema isn't refreshed, the query fails at planning time with: `Schema not found for Iceberg table '<table>'. Run: REFRESH <catalog>=><table>`. Confirm: | ||
| // - Is REFRESH required only the first time, or every time the Iceberg schema changes? |
There was a problem hiding this comment.
Currently it's advised to run REFRESH at any shape change from any side, as that's the view Oxla has and considers during preparing the query. However, I believe @Bixkog is working on performing the REFRESH on all tables automatically on creation of the catalogs, so the tables will be visible immediately after creating the catalogs. The refresh is needed on any subsequent shape change, though, anyway.
95b585a to
6999a6c
Compare
|
|
||
| == Set up the Iceberg query catalogs | ||
|
|
||
| You create three objects, in this order: a storage connection, an Iceberg catalog, and a Redpanda catalog that links to the Iceberg catalog. The storage and Iceberg-catalog options must match the cluster's xref:manage:iceberg/rest-catalog/index.adoc[REST catalog configuration] (endpoint, credentials, region). |
There was a problem hiding this comment.
@Greketrotny if you wanted to continue with default_redpanda_catalog, are you still required to execute CREATE STORAGE, and CREATE ICEBERG CATALOG? Or could you go straight to CREATE TABLE default_redpanda_catalog=>orders (and the storage connection and iceberg catalog are auto-linked under the hood) for an Iceberg-enabled topic?
There was a problem hiding this comment.
Having an iceberg-enabled topic implies, that the CREATE STORAGE, CREATE ICEBERG CATALOG, and CREATE KAFKA CATALOG ... USING ... queries have been invoked by definition. So if the default_redpanda_catalog is provisioned in such a way (I assume it is, when the Enable button next to Oxla cluster is selected when creating a BYOC cluster), then the user can jump right into creating tables, and then running REFRESH, if it's not ran automatically.
There was a problem hiding this comment.
@Greketrotny I reworked this a little so that instead of telling users to execute those CREATEs, the page instead explains that that's done for them: https://deploy-preview-575--rp-cloud.netlify.app/redpanda-cloud/sql/query-data/query-iceberg-topics/#how-the-query-catalogs-are-set-up let me know if this approach doesn't work. cc @mattschumpert
Feediver1
left a comment
There was a problem hiding this comment.
PR Review: SQL: Auto union Iceberg and Redpanda topic (#575)
Files reviewed: 6 .adoc files (466 additions / 17 deletions)
Overall assessment: The cleanest of the SQL GA series. Already engineer-APPROVED. Two broken xrefs are sibling-PR dependencies; same missing What's New entry; no style issues to flag. Ready to land as soon as #571 and #574 do.
What this PR does
Adds Redpanda SQL's Iceberg-bridge query support on the rp-sql integration branch:
modules/sql/pages/query-data/query-iceberg-topics.adoc(new, 123 lines) — how-to that explains the auto-provisioned storage/Iceberg/Redpanda-catalog chain, maps a topic as a SQL table, runs a query that spans live + Iceberg history, and covers schema-divergence rules.modules/reference/pages/sql/sql-statements/create-iceberg-catalog.adoc(new, 210 lines) — reference for the new statement with full options (uri, warehouse, four auth types — none/OAuth2/basic/AWS SigV4, TLS settings) and five worked examples.modules/reference/pages/sql/sql-statements/alter-iceberg-catalog.adoc(new, 40 lines) — modify connection properties of an existing Iceberg catalog.modules/reference/pages/sql/sql-statements/drop-iceberg-catalog.adoc(new, 33 lines) — drop an Iceberg catalog, with the documented constraint that a linked Redpanda catalog must be detached or dropped first.modules/reference/pages/sql/sql-statements/create-redpanda-catalog.adoc(56+ / 16−) — adds the newUSING CATALOGclause, the conditionalpandaproxy_urloption, and a fourth example showing the linked-catalog form.modules/ROOT/nav.adoc— wires the new pages into the nav tree.
Jira ticket alignment
Ticket: DOC-2006 — "Document feature auto union Iceberg and Redpanda topic" (extracted from branch name).
Status: The PR delivers everything the ticket implies — bridge-query how-to + the three new Iceberg catalog SQL statements + USING-CATALOG documentation on the existing Redpanda catalog page. Greketrotny's engineering review addressed the REFRESH semantics and drop-with-link constraint, both of which are now documented correctly. ✓
Critical issues (must fix)
-
Two broken xrefs to sibling-PR targets (verified missing on
rp-sql):File:line xref target Provided by query-iceberg-topics.adoc:13sql:query-data/query-streaming-topics.adoc[]PR #574 (still OPEN) query-iceberg-topics.adoc:21sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL]PR #571 (still OPEN) query-iceberg-topics.adoc:116sql:query-data/query-streaming-topics.adoc[Query streaming topics](Next steps)PR #574 (still OPEN) -
Missing What's New entry. Third PR in the SQL GA series with no entry in
modules/get-started/pages/whats-new-cloud.adoc. As I noted in #571 and #574, a single coordinated "Redpanda SQL: General availability" entry for the whole series would cover this PR alongside the get-started, streaming-query, and OIDC/access pages.
Suggestions (should consider)
-
Filename collision with the existing
manage:iceberg/query-iceberg-topics.adoc. Two pages now share the basenamequery-iceberg-topics.adoc:modules/sql/pages/query-data/query-iceberg-topics.adoc(new, this PR) — nav label: "Query Iceberg-enabled Topics"modules/manage/pages/iceberg/query-iceberg-topics.adoc(pre-existing) — nav label: "Query Iceberg Topics"
They live in different modules and the display labels differ, so navigation isn't actually ambiguous to readers. But it's the kind of overlap that bites later (typo'd xref lands on the wrong page; future grep across
query-iceberg-topicsreturns both). Worth confirming this is intentional and that the two scopes (Redpanda SQL vs. external query engines) are clearly distinct in each page's intro. -
// TODOplaceholder inquery-iceberg-topics.adoc:102:// TODO: Verify with engineering whether there are workload patterns that // reliably trigger longer planning, and document them if so (qa-questions.md #22).- Suggested: Open a follow-up ticket and reference it here (e.g.,
// TODO(DOC-XXXX): ...) so the TODO has an issue trail rather than a freeform pointer to a private QA document.
- Suggested: Open a follow-up ticket and reference it here (e.g.,
Impact on other files
modules/ROOT/nav.adoc✓ — all four new pages added at lines 357 (how-to), 541 (alter), 545 (create), 549 (drop).modules/get-started/pages/whats-new-cloud.adoc❌ — no SQL GA entry (Critical #2).- Cross-page consistency: the new how-to refers to bridge queries planning a union internally; the create-redpanda-catalog reference and create-iceberg-catalog reference describe the same model from their respective directions; create-redpanda-catalog's new "linked catalog" example is consistent with the auto-provisioned
ALTER REDPANDA CATALOG ... USING CATALOGshown in the how-to. No drift detected. ✓ - Cross-component xrefs verified inside this PR's content:
xref:reference:properties/cluster-properties.adoc#iceberg_catalog_type✓xref:manage:iceberg/rest-catalog/index.adoc✓xref:manage:iceberg/about-iceberg-topics.adoc✓xref:sql:connect-to-sql/index.adoc✓xref:reference:sql/sql-statements/create-storage.adoc✓ (already on rp-sql)- All intra-PR xrefs (create/alter/drop/redpanda-catalog statements) ✓
- Only the two sibling-PR xrefs above are unresolved.
- Pre-existing
manage:iceberg/query-iceberg-topics.adocis unchanged by this PR and still in nav at line 433 — no breakage there.
CodeRabbit findings worth considering
None. CodeRabbit's check passed with no actionable findings.
Outstanding review activity (not findings — just status)
- Greketrotny's
APPROVED(May 14 and May 15, two approvals on file). - His follow-up COMMENTED reviews (May 18, May 20) raised three specific points:
- "REFRESH pertains to the schema/shape of the table only" → current diff at lines 93–95 reflects this precisely.
- "User cannot drop an iceberg catalog when there is another Kafka catalog linking" → documented at
drop-iceberg-catalog.adoc:7–8. - "Don't document [this behavior]" on create-redpanda-catalog → Kat replied "Removed", and the current diff is clean.
- Greketrotny's final comment on the how-to was "looks accurate". No outstanding engineer-blocking concerns.
What works well
- Engineering-validated content. Greketrotny iterated on the technical specifics and ended with "looks accurate". The REFRESH guidance, the drop-with-link constraint, and the catalog-provisioning chain were all verified by the implementer.
- "How the query catalogs are set up" section is genuinely useful — it shows the three SQL statements Cloud runs under the hood so administrators can reason about the setup without being asked to run them, and so debugging-via-
DESCRIBEmakes sense. - "Handle schema differences" section documents a real edge case (Iceberg table holding columns the topic schema no longer has, and the resulting planning-time error) with the exact resolution.
- Comprehensive options tables in the new reference pages — covering all four auth types in
CREATE ICEBERG CATALOGplus TLS settings, plus AWS-default-credential-chain behavior for SigV4. - Examples per reference page demonstrate every distinct usage shape — none of the three example pages is a single-snippet page.
Related statementscross-link table at the bottom ofcreate-iceberg-catalog.adocmakes lateral navigation between catalog statements easy.- Frontmatter compliance: description,
:page-topic-type:(how-to / reference), learning objectives observable and measurable, personas (app_developer,data_engineer) correctly scoped to the query-side audience. - No em dashes anywhere in the new content (clean compared to #574).
- All H2+ headings in sentence case; H1s in title case (or all-caps SQL keywords). ✓
- All code blocks have explicit
[source,sql]language tags. Consistent with the SQL module's convention. - CI fully green and the Netlify preview links cover the four new pages.
Final-pass review via /docs-team-standards:pr-review.
Feediver1
left a comment
There was a problem hiding this comment.
@kbatuigas Other than the dependencies on other PRs being merged, and the What's new update, this looks clean. Going to approve with the understanding that the other PRs will be merged, and What's new update.
4444b5c to
1a233e6
Compare
1a233e6 to
82d7ea9
Compare
Description
This pull request adds comprehensive documentation for Redpanda SQL's new support for Iceberg catalogs and bridge queries, enabling users to query both live Redpanda topics and their Iceberg-committed history. It introduces reference and how-to content for creating, altering, and dropping Iceberg catalogs, details the new
USING CATALOGclause for Redpanda catalogs, and provides a step-by-step guide for querying Iceberg-enabled topics.New SQL statement documentation:
CREATE ICEBERG CATALOG,ALTER ICEBERG CATALOG, andDROP ICEBERG CATALOG, including syntax, options (covering authentication and TLS), and usage examples. [1] [2] [3]Enhancements to Redpanda catalog documentation:
USING CATALOGclause forCREATE REDPANDA CATALOG, which links a Redpanda catalog to an Iceberg catalog for bridge queries.pandaproxy_urloption, required when usingUSING CATALOG, and provided an example of creating a linked catalog. [1] [2]How-to guide for querying Iceberg-enabled topics:
Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 18 May
Page previews
Redpanda SQL > Query Data > Query Iceberg topics
Reference > Redpanda SQL Reference > Statements
CREATE ICEBERG CATALOG
ALTER ICEBERG CATALOG
DROP ICEBERG CATALOG
CREATE REDPANDA CATALOG > Create catalog linked to Iceberg catalog
Checks